Add release-tracker-workflow pipeline for Shaman-based RC testing by deepssin · Pull Request #2539 · ceph/ceph-build

deepssin · 2026-02-17T11:54:30Z

This adds a Jenkins pipeline to automate RC testing for release trackers: resolve or pass a Ceph SHA1, wait for it on Shaman, schedule teuthology suites (in parallel), aggregate results, and optionally post to Redmine and send email. There is no build step; the pipeline assumes the SHA1 is already available on shaman.
Note: The job needs to run on a teuthology agent (that means we need to add teuthology-node as an agent). The pipeline is parameterized (e.g. AGENT_LABEL, teuthology paths, Paddles/Pulpito URLs) so it can be adapted to different environments.

This is meant as a starting point for discussion—defaults and layout may need to change depending on how we want to run this in production (e.g. which Jenkins, which agents, credentials). Happy to adjust based on feedback.

djgalloway · 2026-02-17T15:35:17Z

release-tracker-workflow/build/Jenkinsfile

+        string(name: 'CEPH_REPO', defaultValue: 'https://github.com/ceph/ceph.git')
+        string(name: 'CEPH_BRANCH', defaultValue: 'main')
+        string(name: 'CEPH_SHA1', defaultValue: '', description: 'Optional: Ceph commit SHA1 to use. If set, run on this SHA1 (must exist on Shaman). Empty = resolve from branch tip.')
+        string(name: 'RELEASE_VERSION', defaultValue: '')


Is this a release version of Ceph? Like a tag?

yes , that's right.

Can you put an example in the description please

dmick · 2026-02-17T18:24:20Z

What is a release tracker?

deepssin · 2026-02-18T15:05:32Z

What is a release tracker?

The tracker meant for a Ceph release like below -
https://tracker.ceph.com/issues/72316
https://tracker.ceph.com/issues/73906

djgalloway · 2026-02-18T19:35:33Z

release-tracker-workflow/build/Jenkinsfile

+            steps {
+                script {
+                    def w = "${env.PIPELINE_DIR}/scripts/wait_for_shaman_sha1.py"
+                    if (fileExists(w)) sh "python3 ${w} --branch ${params.CEPH_BRANCH} --sha1 ${env.SHA1} --timeout ${params.SHAMAN_WAIT_TIMEOUT} --interval ${params.SHAMAN_WAIT_INTERVAL}"


Let's say Ubuntu builds always take 1 hour and CentOS builds always take 2 hours. How does this handle shaman saying, "Yes, Ubuntu is done?" Won't this proceed with scheduling a run for CentOS and Ubuntu because we're not checking for all distros?

@djgalloway Good question. The script already waits for all configured platforms. By default it checks both ubuntu-jammy-default and centos-9-default and only proceeds when the SHA1 is present on all of them (all(...) on line 46). The --platform argument can be customized if different distros are needed.

djgalloway · 2026-02-27T15:29:41Z

I guess I'm fine with this conceptually. Has it been tested anywhere? I'd like to see it working.

deepssin · 2026-03-05T07:20:15Z

I guess I'm fine with this conceptually. Has it been tested anywhere? I'd like to see it working.

I am working on getting logs for this, will share ASAP

Signed-off-by: deepssin <deepssin@redhat.com>

deepssin · 2026-03-16T13:16:20Z

Here are logs from my jenkins instance - release-testing.log
it updated tracker - https://tracker.ceph.com/issues/75290 . Trying to get the soko04 machine added to ceph jenkins and access to test this in upstream jenkins.

djgalloway · 2026-03-16T14:11:02Z

I just connected soko04 to Jenkins - https://jenkins.ceph.com/computer/10%2E20%2E192%2E14%2Bsoko04/

deepssin · 2026-03-17T13:30:58Z

@djgalloway, thanks! -I tried to test the release-tracker pipeline on soko04 it's failing because teuthology hits Paddles over HTTPS and the Sepia cert isn’t trusted for the jenkins-build user. can we add the Sepia CA to the trust store on soko04 so jenkins-build can reach https://paddles-paddles.apps.pok.os.sepia.ceph.com? https://jenkins.ceph.com/view/all/job/preserve-release-tracker-workflow/13/console

djgalloway · 2026-03-17T19:59:45Z

can we add the Sepia CA to the trust store on soko04

Hm, it actually already is.

dgalloway@soko04:~$ openssl s_client   -connect paddles-paddles.apps.pok.os.sepia.ceph.com:443   -servername paddles-paddles.apps.pok.os.sepia.ceph.com </dev/null   | openssl verify -CAfile /etc/ssl/certs/ca-certificates.crt
depth=1 CN = ingress-operator@1765490096
verify return:1
depth=0 CN = *.apps.pok.os.sepia.ceph.com
verify return:1
DONE
stdin: OK

Does python need to be informed of this somehow?

dmick · 2026-03-17T22:34:42Z

fwiw, this works too:

cat testsock.py

import socket
import ssl

c = ssl.create_default_context()
h = 'paddles-paddles.apps.pok.os.sepia.ceph.com'
with socket.create_connection((h,443)) as sock:
    with c.wrap_socket(sock, server_hostname=h) as ssock:
        print(ssock.version())

strace shows that it opens /etc/ssl/certs/ca-certificates.crt

deepssin · 2026-03-18T14:01:32Z

Python’s ssl module uses the system CA store (/etc/ssl/certs/ca-certificates.crt), so your test works. The problem is that the requests library (used by teuthology) uses certifi’s bundled certs by default, not the system store. Certifi’s bundle doesn’t include the Sepia CA.

one way is to point requests at the system certs:

export REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt

I've added this as a param in my test pipeline and was able to trigger a teuthology run https://jenkins.ceph.com/view/all/job/preserve-release-tracker-workflow/29/console , but I'm not sure parametrizing the CA bundle path is the right approach. might be we can have this a global variable set on the machine . Thoughts?

djgalloway · 2026-03-18T14:52:19Z

I put export REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt in jenkins-build user's .bashrc on soko04. The agent will likely need to be restarted for it to take effect, however.

dmick · 2026-03-18T17:56:49Z

interesting. On Ubuntu, the system "python3-certifi" library contains the library modified to use /etc/ssl:

$ cat /usr/lib/python3/dist-packages/certifi/core.py
"""
certifi.py
~~~~~~~~~~

This module returns the installation location of
/etc/ssl/certs/ca-certificates.crt or its contents.
"""

DEBIAN_CA_CERTS_PATH = '/etc/ssl/certs/ca-certificates.crt'


def where() -> str:
    return DEBIAN_CA_CERTS_PATH


def contents() -> str:
    with open(where(), "r", encoding="ascii") as data:
        return data.read()

Perhaps we should install it with the system package. Perhaps there are similar things for el systems.

deepssin requested review from djgalloway, dmick, neha-ojha and zmc February 17, 2026 11:54

djgalloway reviewed Feb 17, 2026

View reviewed changes

djgalloway reviewed Feb 18, 2026

View reviewed changes

Add release-tracker-workflow pipeline for Shaman-based RC testing

ea0f0da

Signed-off-by: deepssin <deepssin@redhat.com>

deepssin force-pushed the release-tracker-workflow branch from 24ec0e7 to ea0f0da Compare March 16, 2026 13:10

Conversation

deepssin commented Feb 17, 2026

Uh oh!

djgalloway Feb 17, 2026

Choose a reason for hiding this comment

Uh oh!

deepssin Feb 18, 2026

Choose a reason for hiding this comment

Uh oh!

djgalloway Feb 27, 2026

Choose a reason for hiding this comment

Uh oh!

dmick commented Feb 17, 2026

Uh oh!

deepssin commented Feb 18, 2026

Uh oh!

djgalloway Feb 18, 2026

Choose a reason for hiding this comment

Uh oh!

deepssin Feb 25, 2026

Choose a reason for hiding this comment

Uh oh!

djgalloway commented Feb 27, 2026

Uh oh!

deepssin commented Mar 5, 2026

Uh oh!

deepssin commented Mar 16, 2026

Uh oh!

djgalloway commented Mar 16, 2026

Uh oh!

deepssin commented Mar 17, 2026

Uh oh!

djgalloway commented Mar 17, 2026

Uh oh!

dmick commented Mar 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

deepssin commented Mar 18, 2026

Uh oh!

djgalloway commented Mar 18, 2026

Uh oh!

dmick commented Mar 18, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

dmick commented Mar 17, 2026 •

edited

Loading